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SUBSTITUTE SPECIFICATION 



AUTOMATED EVALUATION SYSTEM AND PROGRAM 
[0001] TECHNICAL FIELD 

[0002] The present invention relates to an automated evaluation 

system that automatically evaluates an application program that 
operates on a target system such as a microcomputer according to 
input events such as key inputs and reference outputs such as output 
pictures and the like with respect to the input events. 
[0003] BACKGROUND TECHNOLOGY 

[0004] In recent years, microcomputers have been widely used 

and implemented in a variety of apparatuses such as home electric 
appliances. A microcomputer has a built-in ROM (Read Only Memory) 
in which an application program is stored, in order to operate 
according to the specification of an apparatus in which the 
microcomputer is implemented or peripheral apparatuses therefor. 
Also, liquid crystal display apparatuses (hereinafter referred to as 
LCDs "Liquid Crystal Displays") are implemented in a variety of 
apparatuses such as home electric appliances. Therefore, when the 
user inputs an input event such as a key input, a microcomputer 
outputs to a LCD an output picture corresponding to the input event 
according to an application program. In this respect, in the event of 
confirming the operation of the application program by the 
microcomputer, output results for numerous input events according to 
the specification of an apparatus in which the microcomputer is 
implemented must be confirmed. 

[0005] Accordingly, when a microcomputer is developed, not only 

development of its hardware is important, but also development of an 
application program is important. For development of an application 
program, an InCircuit Emulator (hereinafter referred to as ICE 
"InCircuit Emulator") or the like is used. The ICE can emulate an 
operation of an application program on a target board. As described 
above, for confirming the operation of an application program, there 
are numerous input events that are to be confirmed. Therefore, when 
the operator directly inputs input events using the ICE, it takes a long 
time to input them, and there is a possibility that operator may make 
input errors. Also, when the operator confirms input events and 
emulated results by picture outputs, such a work takes a long time and 
there is a possibility that the operator may make confirmation errors. 
[0006] Furthermore, in order to evaluate the application 

program by confirming its operation, the same events are repeatedly 
evaluated in order to improve the evaluation accuracy. Therefore, in 
order to confirm the operation of the application program more 
efficiently at a higher accuracy, an automated evaluation system that 
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can automatically and repeatedly input numerous input events and 
that can automatically evaluate output results corresponding to the 
input events is used. 

[0007] Fig. 4 shows a structure for performing an automated 

evaluation of an application program of a conventional microcomputer. 
An automated evaluation system 50 is composed on a personal-purpose 
computer (hereinafter referred to as a personal computer) PC, and is 
capable of accessing a disk apparatus DU of the personal computer PC. 
Furthermore, the automated evaluation system 50 is equipped with an 
emulation expansion board 50a in order to enter input events in a 
target board TB. On the other hand, an ICE 51 is connected to the 
personal computer PC through an RS 232 and also to the target board 
TB. Also, a debugger 52 is implemented on the personal computer PC, 
such that, while the ICE 51 is emulating an application program AP, 
the debugger 52 can perform debugging. In the event of automatic 
evaluation of the application program AP, first, the operator prepares 
an input event file IF that stores a plurality of input events and a 
reference output file OF that stores reference output results 
corresponding to the input events. Then, when the automated 
evaluation system 50 is started, the automated evaluation system 50 
reads an input event stored in the input event file IF and a reference 
output result stored in the reference output OF from the disk drive 
DU. Then, the automated evaluation system 50 enters the input event 
through the emulation expansion board 50a to the target board TB. 
[0008] Then the ICE 51 emulates the operation of the application 

program AP according to the input event, and outputs an output result 
to the automated evaluation system 50. Then, the automated 
evaluation system 50 compares the output result and the reference 
output result to determine whether or not the output result is correct, 
and outputs a determination result to a result log file or the like. 
Furthermore, the automated evaluation system 50 automatically 
performs the input, comparison and determination steps repeatedly for 
a plurality of input events and automatically evaluates the application 
program AP. 

[0009] However, in the automated evaluation by the automated 

evaluation system 50 and ICE 51, data is transferred between the 
automated evaluation system 50 and the target board TB, and 
therefore a function for automated evaluation needs to be 
implemented. In other words, the automated evaluation system 50 
needs the emulation expansion board 50a in order to convert input 
events to data for emulation. Furthermore, the target board TB is 
provided with an expansion board interface TBa implemented therein 
in order to receive input events and a personal computer interface TBb 
for outputting output results to the personal computer PC. Also, an 
automated evaluation system interface program IP is implemented in 
the application program AP in order to transfer data to a VRAM (Video 
Random Access Memory) for analyzing the input events from the 
automated evaluation system 50 or outputting output results to the 
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personal computer PC. As a result, the application program AP to be 
evaluated by the automated evaluation is different from an application 
program that is actually implemented in the microcomputer, and the 
automated evaluation system interface program IP adds a load to a 
process that is to be performed by the microcomputer. In this manner, 
when a function for automated evaluation is added, unstable factors in 
the evaluation increase and the quality of the application program AP 
to be evaluated lowers. 

[00010] Therefore, it is an object of the present invention to 
provide an automated evaluation system that can retain the quality of 
an application program and also maintain the stability in the 
automated evaluation. 

[00011] SUMMARY OF THE INVENTION 

[00012] An automated evaluation system in accordance with the 
present invention to solve the problems described above is provided, 
wherein the automated evaluation system automatically evaluates an 
application program to be operated on a target system according to a 
given input event by comparing a reference output result 
corresponding to the given input event and a simulation result 
generated by a simulator in response to the given input event. An 
evaluator is accessible to a simulation apparatus that simulates an 
operation of the application program and generates the simulation 
result corresponding to the given input event. The reference output 
result is then compared to the corresponding simulation result to 
thereby perform an automated evaluation. 

[00013] In accordance with the automated evaluation system that 
is accessible to the simulation apparatus, an automated evaluation can 
be performed using a simulation result corresponding to an input 
event. In this instance, the simulation apparatus performs a normal 
simulation of an operation of the application program based on the 
input event. Accordingly, there is no need to implement a function for 
automated evaluation in the application program. 
[00014] Furthermore, the automated evaluation system is 
equipped with a memory device that is accessible by both the 
simulation apparatus and the automated evaluation system, and 
characterized in that the simulation result stored in the memory 
device is compared with the reference output result. 
[00015] The memory accessible by both automated evaluation 
system and the simulation apparatus further enables the latter two 
components to communicate with one another. 

[00016] As a result, in an automated evaluation by the automated 
evaluation system and the simulation apparatus, a function for 
automated evaluation does not need to be implemented to perform the 
automated evaluation. 

[00017] Furthermore, the simulation apparatus and the 
automated evaluation system can be realized using the same 
computer. 
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[00018] According to the automated evaluation system described 
above, the automated evaluation system and the simulator can 
communicate with each other regarding simulation results through a 
memory device provided in the computer. 

[00019] The present invention also provides a program that 
renders a computer capable of performing an automated evaluation of 
a application program to be operated on a target system according to a 
given input event by comparing a reference output result 
corresponding to the given input event with a simulation result 
generated in response to the given input event. 

[00020] The program is accessible to a simulation apparatus that 
simulates an operation of the application program, and compares a 
simulation result generated by the simulation apparatus in response to 
the given input event with the corresponding reference output result to 
thereby render the computer able to perform an automated evaluation 
of the application program. 

[00021] In accordance with the program, which is made accessible 
to the simulation apparatus, an automated evaluation can be 
performed using a simulation result for an input event of the 
simulation apparatus. In this instance, the simulation apparatus 
performs a normal simulation of an operation of the application 
program based on the input event. Accordingly, there is no need to 
implement a function for automated evaluation in the application 
program, and the automated evaluation system can be realized by the 
computer while retaining the quality of the application program. 
[00022] Also, the program in accordance with the present 
invention is characterized in that it is accessible to the simulation 
apparatus through an application programming interface implemented 
in an operating system of the computer. 

[00023] According to the program described above, the automated 
evaluation system and the simulation apparatus can communicate 
using a function implemented in the operating system of the computer 
that executes the program. 

[00024] BRIEF DESCRIPTION OF THE DRAWINGS 
[00025] Fig. 1 shows an exemplary structure of a personal 
computer in which an automated evaluation system and a simulation 
apparatus are operated in accordance with an embodiment of the 
invention. 

[00026] Fig. 2 is an illustration showing access and 
communication between an automated evaluation system, a simulator, 
and a memory (RAM) in accordance with an embodiment of the present 
invention. 

[00027] Fig. 3 shows an example of input events and reference 
output results, wherein Fig. 3 (a) shows key inputs (keystroke) input 
events, Fig. 3 (b) shows a reference output picture on an LCD before 
any input event is executed, and Fig. 3 (c) shows a reference output 
picture on the LCD for the key inputs shown in Fig. 3 (a). 
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[00028] Fig. 4 shows a conventional structure for performing an 
automated evaluation of an application program of a microcomputer. 
[00029] DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[00030] An automated evaluation system in accordance with an 
embodiment of the present invention is described below with reference 
to the accompanying drawings. 

[00031] The automated evaluation system in accordance with the 
present invention automatically evaluates an application program to 
be operated on a target system by comparing simulation results with 
respective reference results for respective input events. In doing so, a 
simulation apparatus performs a normal process, in which a 
simulation for an input event is performed and a result of the 
simulation is output. Therefore, a function for an automated 
evaluation does not need to be implemented in the application 
program. 

[00032] It is noted that the target system is operated based on an 
application program. The target system includes, for example, a 
microcomputer. Also, input events to be applied are set according to 
the specification of an apparatus in which the target system is 
implemented, and input events to be applied vary depending on input 
devices of the target system apparatus. Input events to be applied are, 
for example, key inputs (e.g., keystrokes), voice inputs, and the like. 
Reference output results are normal outputs of the target system in 
response to input events, and are set for the specification of an 
apparatus in which the target system is implemented. Reference 
output results to be used vary depending on output devices of an 
apparatus in which the target system is implemented. Reference 
output results to be used include, for example, picture outputs, voice 
outputs, and the like. 

In accordance with an embodiment of the present invention, the 
automated evaluation system is composed as an automated evaluation 
system that performs an automated evaluation by loading an 
automated evaluation program from a storage media onto a personal 
computer and rendering the personal computer capable of executing an 
operation in accordance with the automated evaluation program. 
Furthermore, the automated evaluation system accesses a disk 
apparatus in communication with the personal computer to obtain 
input events and reference output results. Also, in accordance with the 
present embodiment, the simulation apparatus is composed as a 
simulator that performs a simulation by loading a simulation program 
from a storage media onto the personal computer and rendering the 
personal computer capable of executing an operation by the simulation 
program. It is noted that the automated evaluation system and the 
simulator may be composed within the same personal computer. Also, 
in accordance with the present embodiment, the target system is a 
microcomputer. Furthermore, in accordance with the present 
embodiment, the microcomputer can be implemented in an apparatus 
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that can execute key (button) inputs from the exterior and picture 
outputs on an LCD, such as, for example, a gaming machine, a watch, 
a data bank and the like. 

[00033] First, referring to Fig. 1, the entire structure of an 
automated evaluation system 1 (e.g., a validator) and a simulator 2 
(e.g., an emulator) is described. 

[00034] In the automated evaluation system 1, an automated 
evaluation program is read from a storage media that stores the 
automated evaluation program to a disk apparatus DU of a personal 
computer PC, where the automated evaluation program is loaded onto 
a RAM (Random Access Memory) (not shown). The automated 
evaluation program is then executed by a central processing apparatus 
CP to perform an automated evaluation. The disk apparatus DU may 
be an apparatus that can read to and write from a storage media such 
as a floppy disk, an optical disk and the like. Also, the automated 
evaluation system 1 reads an input event stored in an input event file 
IF in the disk apparatus DU, and transfers the input event to the 
simulator 2. Furthermore, the automated evaluation system 1 reads in 
a reference output result that is stored in a reference output file OF in 
the disk apparatus DU and compares the same with input-event-based 
simulation result generated by the simulator 2 to perform an 
automated evaluation. It is noted that the automated evaluation 
system 1 may store an evaluation result in the disk apparatus DU as a 
result log file, and outputs a picture on a display DP. 
[00035] In the simulator 2, a simulation program is read from a 
storage media that stores the simulation program to the disk 
apparatus DU of the personal computer PC, where the simulation 
program is loaded on the RAM (not shown). The simulation program is 
then executed by the central processing apparatus CP to perform a 
simulation. Also, the simulator 2 reads an application program AP of 
the microcomputer from the disk apparatus DU, and simulates an 
operation by the application program AP. 

[00036] It is noted that, when an input event is transmitted by 
the automated evaluation system 1, the simulator 2 performs a 
simulation of the operation by the application program AP based on 
the input event. Then, the simulator 2 stores a result of the simulation 
in a RAM 10 of the personal computer PC (see Fig. 2). 
[00037] Also, the simulator 2 can be externally operated by a 
keyboard KB corresponding to key inputs of the implemented 
apparatus, and outputs an LCD picture on the display DP 
corresponding to an LCD picture output of the implemented apparatus. 
[00038] The simulator 2 is in communication with a debugger 3, 
such that, while a simulation is performed, the application program AP 
can be debugged. With the debugger 3, a debug program is read from a 
storage media that stores the debug program to the desk apparatus 
DU of the personal computer PC, where the program is loaded onto a 
RAM (not shown). The program is then executed by the central 
processing apparatus CB to perform a debugging. The debugger 3 can 
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start and stop the simulator 2, refers to data or rewrite data on the 
simulator 2. Furthermore, the debugger 3 can execute the application 
program AP at each step or break it. 

[00039] Next, the input event file IF and the reference output file 
OF are described below. The input event file IF and the reference 
output file OF can be created using an input event data creating 
function or a reference data creating function of the automated 
evaluation system 1, or may be created in advance by an editor. 
[00040] First, the case in which the input event file IF is created 
by the automated evaluation system 1 is described. First, keys of the 
apparatus in which the microcomputer is implemented are assigned to 
keys in the keyboard KB by the automated evaluation system 1. Then, 
the user prepares multiple input events according to the specification 
of the implemented apparatus and inputs them one by one using the 
keys. Then, the automated evaluation system 1 confirms the key type 
for each of the input events and the input order as input event data. 
Finally, the automated evaluation system 1 stores the event data for 
the entire input events in the input event file IF. It is noted that the 
input event file IF is stored in a storage media with an optional 
filename being added and set in a state such that it can be read from 
the disk apparatus DU. Incidentally, the input event file IF can be 
modified according to changes in the specification of the 
microcomputer, changes in the specification of the implemented 
apparatus, changes in the evaluation contents, or the like. For 
example, as shown in Fig. 3 (a), let us assume that a key operation is 
conducted for one input event 20, in which a key [A] is depressed, a key 
[B] is depressed and a key [C] is depressed in this order. In this case, 
the input event 20 is stored in the input event file IF as input event 
data defining key types of the keys A, B and C and the key input order. 
[00041] Next, the case in which the reference output file OF is 
created by the automated evaluation system 1 is described. Since the 
reference output file OF stores reference data in which each one 
reference output result corresponds to each one input event, it is 
created corresponding to the creation of the input event file IF. Each 
time the user inputs an input event by operating a key, the automated 
evaluation system 1 transfers the key input to the simulator 2 as an 
input event. Then, the simulator 2 performs a simulation for the key 
input, and displays a simulation result on the display DP. Upon 
display, the user confirms the displayed content on the display DP, and 
confirms it as a reference output result if the content is correct. In this 
case, since an output device of the implemented apparatus is an LCD, 
the reference output result (reference data) includes image data to be 
displayed on an LCD and position data for displaying on the LCD. It is 
noted that when an application program AP is under development, a 
reference output file OF for upgrading the version of the application 
program AP may be created. In this instance, the reference output file 
OF can add corrections of bugs or changes in the specification and the 
like to the automated evaluation items of the application program AP 
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so that the automated evaluation including the changes in the 
application program AP can be performed. Alternatively, after the 
user inputs an input event by a key operation, image data for 
displaying on the LCD may be formed as a reference output result by a 
reference-data-creating editor of the automated evaluation system 1. 
[00042] Finally, the automated evaluation system 1 stores 
reference data for the entire reference output results in the reference 
output file OF. It is noted that the reference output file OF is stored in 
a storage media with an optional filename being added, and is set in 
the state such that it can be read from the disk apparatus DU. The file 
name of the reference output file OF is described in the input event file 
IF, and the reference output file OF is read according to the input 
event file IF. Therefore, the reference output file OF is modified 
according to the input event file IF. For example, as shown in Fig. 3 
(b), let us assume that, before the input event 20 is inputted, a 

reference output picture 21 displays [ ] on its left upper end on the 

LCD. 

[00043] Then, when the keys are operated for the input event 20 

as shown in Fig. 3 (a), a reference output picture 22 displays [ABC ] 

starting from the left upper end to the right on the LCD as a reference 
output result 23. In this case, the reference output result 23 is stored 

as reference data including image data for displaying [ABC ] on the 

LCD and position data for displaying the image on the LCD in the 
reference output file OF. 

[00044] Next, referring to Fig. 2, operations of the automated 
evaluation system 1 and the simulator 2 that take place when an 
automatic evaluation is performed are described. 

[00045] The automated evaluation system 1, when started by the 
user, loads input event data ID stored in the input event file IF from 
the disk apparatus DU onto the personal computer PC. It is noted that 
the input event file IF is designated by its file name by the user. When 
the input event file IF is loaded, the automated evaluation system 1 
loads reference data RD stored in the reference output file OF having 
the filename described in the input event file IF onto the personal 
computer PC. 

[00046] Then, the automated evaluation system 1 transfers one 
input event from the input event data ID to the simulator 2. For 
transmission of the input event, an API [Application Programming 
Interface] command of the OS [Operating System] on the personal 
computer is used. For example, when the OS is Windows, one API 
command, i.e., FindWindow is used to obtain a window handle of the 
simulator 2. Then, another API command, PostMessage is used to 
transmit the one input event among the input event data ID to the 
window handle. In other words, the automated evaluation system 1 
and simulator 2 can communicate with each other with respect to 
input events by using API commands. It is noted that the automatic 
evaluation system 1 and the simulator 2 use the functions that are 
provided in the operating system, such as, API commands. Therefore, 
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a special function for transmitting input events does not need be 
added. 

[00047] Each time an input event is transmitted, the simulator 2 
simulates an operation to be performed by the application program AP 
based on the input event. Then, the simulator 2 temporarily stores 
LCD display image data and position data as a simulation result in the 
RAM 10 for displaying on the display DP. Also, the simulator 2 
displays the LCD display image data stored in the RAM 10 on the 
display DP. 

[00048] The process performed by the simulator 2 is the same as 
the normal process in which an operation by the application program 
AP is simulated, and a special process is not performed for the 
automated evaluation. Therefore, the application program AP that is 
the same as an application program actually mounted on a 
microcomputer can be used. It is noted that the RAM 10 is a built-in 
RAM in the personal computer PC, and is a RAM that is commonly 
shared by the automated evaluation system 1 and the simulator 2. 
Therefore, the RAM 10 can be accessed by the automated system 1 and 
the simulator 2. In other words, the automated evaluation system 1 
and the simulator 2 can communicate with each other with respect to 
simulation results through the RAM 10. Since the automated 
evaluation system 1 and the simulator 2 both use the RAM 10 of the 
personal computer PC, no special function needs to be added for 
referring to the simulation result. It is noted that the RAM 10 may be 
a VRAM of the personal computer. 

[00049] It is noted that, in the present embodiment, the RAM 10 
corresponds to a storage device. 

[00050] After the simulation, the automated evaluation system 1 
refers to the simulation result stored in the RAM 10. Then, the 
automated evaluation system 1 compares the simulation result and the 
reference data (RD), that is, the reference output result (the LCD 
display image data and the position data) corresponding to the loaded 
input event that was transmitted to the simulator 2. The automated 
evaluation system 1 makes a determination according to whether or 
not both of the results concur with each other, to thereby evaluate the 
operation of the application program with respect to the input event. 
Furthermore, the automated evaluation system 1 stores the 
determination result in a result log file. The result log file may store 
all of the determination results, or may store determination results 
only when the simulation results differ from the reference output 
results. It is noted that the automated evaluation system 1 may 
display the simulation results and reference output results side-by-side 
on the display DP, so that the user can confirm them. Also, the 
automated evaluation system 1 may display determination results on 
the display DP. 

[00051] Each time an evaluation for one input event is completed, 
the automated evaluation system 1 repeats the process described above 
for the next input event stored in the input event data ID to thereby 
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carry out the automated evaluation. When the evaluation of all of the 
input events in the input event data ID is completed, the automated 
evaluation system 1 stores the evaluation results in a log file in a 
storage media such as a hard disk or the like according to the 
instruction by the user, and completes the automated evaluation. 
[00052] According to the automated evaluation system 1, the 
automated evaluation system 1 and the simulator 2 can communicate 
through API commands and the RAM 10, which is accessible to both 
the system 1 and the simulator 2. Therefore, for transferring input 
events from the automated evaluation system 1 to the simulator 2, or 
for referring by the automated evaluation system 1 to simulation 
results of the simulator 2, special functions do not need to be added to 
the automated evaluation system 1 or the simulator 2. Furthermore, 
the application program AP does not need to have a function for 
automated evaluation implemented therein, and can be the same as an 
application program that is actually mounted on the microcomputer. 
[00053] The present invention is not limited to the embodiments 
described above; rather, many modifications can be made within the 
scope of the appended claims. 

[00054] For example, API commands and the RAM 10 are used 
such that the automated evaluate system and the simulation 
apparatus can communicate with each other. However, other 
devices/functions can be used to facilitate such communication. 
[00055] Also, the automated evaluation system and the simulator 
are composed on the same personal computer. However, they can be 
composed on other electronic processors such as workstations. 
[00056] Since the automated evaluation system in accordance 
with the present invention described above is capable of 
communicating with the simulation apparatus, an automated 
evaluation can be performed using simulation results for input events 
of the simulation apparatus. In this instance, the simulation 
apparatus only performs a normal simulation for the input events. 
Accordingly, there is no need to implement a function for automated 
evaluation in the application program. Accordingly, the quality of the 
application program is equivalent to that of an application program 
mounted on a target system. 

[00057] Also, the automated evaluation system in accordance with 
the present invention is equipped with a storage device that can be 
accessed by the automated evaluation system and the simulation 
apparatus, such that the automated evaluation system and the 
simulation apparatus can communicate. Therefore, in an automated 
evaluation by the automated evaluation system and the simulation 
apparatus, a function for automated evaluation does not need to be 
implemented. As a result, the stability in the automated evaluation 
can be maintained. 
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